Processing a stream of pixel values in an image processing system using pixel value subset groups

ABSTRACT

Data processing systems (e.g. image processing systems) and methods are provided for processing a stream of data values (e.g. pixel values). A processing module implements processing of a particular pixel value of the stream by operating on a particular subset of pixel values of the stream, by: classifying each of the pixel values within the particular subset into a group of a set of groups; processing the particular pixel value using one or more of the pixel values of the particular subset in dependence on the classification of the pixel values of the particular subset into the groups, wherein said processing the particular pixel value comprises performing, in a consolidated operation, multiple processing functions which depend upon the classification of pixel values of the particular subset into the groups; and outputting the processed particular pixel value.

BACKGROUND

There are many different types of data processing systems which canprocess a stream of data values. For example, an image processing systemmay be used as a camera pipeline to process pixel values originatingfrom image sensors in a camera to provide a set of processed pixelvalues representing a captured image.

An image processing system, such as a camera pipeline, can be arrangedto receive a stream of pixel values (e.g. in raster scan order, or anyother predetermined order, such as Boustrophedon order) and performmultiple processing functions on the pixel values in order to determineprocessed pixel values which can then be output, e.g. for display on adisplay or for storage in a memory or for transmission to anotherdevice, e.g. over a network such as the Internet. FIG. 1a illustrates animage processing system 104 which is arranged to receive image data(e.g. pixel values) from an image sensor 102. As is known in the art,the image sensor 102 can include an array of sensors (e.g.charge-coupled devices (CCDs) or complementary metal-oxide-semiconductor(CMOS) sensors) which can sense incident light at respective pixelpositions to determine raw image data (e.g. unprocessed pixel values)which can be fed into the image processing system 104. In this way theimage sensor 102 captures light and converts it into electrical signals(e.g. image data in the form of pixel values) to be processed in orderto generate an image. The image processing system 104 comprises a numberof distinct processing modules (e.g. FIG. 1 illustrates k processingmodules 106 ₁, 106 ₂, 106 ₃, . . . 106 _(k)) which implement respectiveprocessing functions on the image data to determine the processed imagedata which represents an image and which is output from the imageprocessing system 104. For example, the image processing system 104 maymodify the pixel values (e.g. to improve the perceptual quality of theimage) using functions such as filtering and sharpening. Other functionsof image processing may be to compress or encode image data intoparticular formats, such as the Joint Photographic Experts Group (JPEG)format or the Moving Picture Experts Group (MPEG) format.

An image processing system may be implemented in a pipeline arrangement,whereby multiple processing modules of the image processing systemperform respective functions on pixel values representing an image. Theprocessing modules 106 ₁ to 106 _(k) operate on pixel values in turn, ina sequence, and may operate independently of each other. There are manydifferent processing functions which may be performed by respective onesof the processing modules 106. Some examples of the different processingfunctions which may be applied by a processing module are: adjustinggains applied to the pixel values, adjusting the white balance of thepixel values, detecting defective pixels within the image, correctingdefective pixels, filtering (e.g. denoising), sharpening andde-mosaicing, to give just a few examples. A person skilled in the artwould be aware of many different functions that may be performed bydifferent processing modules within the image processing system 104. Theexact processing modules to implement, their configuration and the orderin which they are implemented in the image processing system 104 (e.g.for use in a camera pipeline) are design choices which are made when theimage processing system 104 is designed.

Some of the processing modules operate on individual pixel values,without needing to consider other pixel values. For example, theadjustment of gains and the adjustment of the white balance can beperformed for a pixel value without considering other pixel values. Incontrast, some of the other processing modules operate on a kernel (i.e.a block or subset) of pixel values. For example, 2D filtering operationssuch as denoising, defective pixel detection and correction operations,sharpening operations and de-mosaicing operations operate on a kernel ofpixel values in order to determine how to process a particular pixelvalue. For example, FIG. 1b shows a kernel 120 of pixel values of animage. The kernel 120 includes forty nine pixel values, arranged as a7×7 block of pixel values, centred on a pixel value 122, which is acurrent pixel value being processed. Since, in a camera pipeline thepixel values are received row-by-row (e.g. in raster scan order), inorder for a processing module 106 to have access to a block of pixelvalues including pixel values from multiple rows, a line store modulemay be implemented in the image processing system 104 prior to aprocessing module 106. Each row of pixel values may for example includehundreds or thousands of pixel values. A line store module can storemultiple rows of pixel values, and can provide a contiguous block ofpixel values (e.g. the block 120) to the processing module 106 at atime. A line store module is a large block to implement in hardwarebecause it typically needs to be able to store thousands of pixel valuesat a given time.

Many of the processing functions which may be implemented by theprocessing modules 106 on blocks of pixel values may involve performingsome sort of edge analysis of the image represented by the pixel valueswithin a block. Some examples of edge detection algorithms (such asSobel edge detection and Canny edge detection) use differential methods.It is noted that the detection of edges within a kernel issuper-linearly hard, e.g. the Canny algorithm has a complexity

(n² log n) for a kernel containing n pixels. For example, a denoisingfiltering function may apply spatial smoothing to the pixel valueswithin a kernel. Denoising is a known process in image processing. Forexample, the pixel value 122 may be filtered based on other pixel valueswithin the kernel 120, e.g. the pixel value 122 may be smoothed so thatit is based on a weighted sum of a plurality of the pixel values withinthe kernel 120 (e.g. where the weights depend upon the distance betweenthe pixel value 122 and the pixel value being summed). In a simpleexample, the pixel value 122 may be based on a mean of a plurality ofthe pixel values within the kernel 120. Spatial smoothing helps toreduce the appearance of random noise in the image. However, it can beperceptually detrimental to an image to smooth pixel values over adistinct edge in an image. This can have the effect of blurring the edgein the image. So a processing module 106 implementing a denoising filterwill attempt to determine the presence of an edge within the kernel 120.There are many different algorithms for analysing a kernel of pixelvalues to determine an edge. The example shown in FIG. 1b shows an edge124 between two materials (A and B) in the scene being imaged, where theedge 124 passes through the kernel 120. The pixels in the kernel 120have been labelled as either “A” or “B” to represent which material isvisible in the image at the respective pixel position. The denoisingapplied to pixel value 122 may be implemented as a weighted sum of (oran average of) only the pixel values labelled A (and not including thepixel values labelled B), where the weights may depend upon the distanceto the pixel 122. In this way the processing module 106 can perform abilateral filtering process to implement the denoising.

Similarly, a processing module 106 which implements sharpening willidentify the edge 124 in the kernel 120 and will apply sharpening topixel values near the edge to increase the differences in pixel valueson opposing sides of the edge 124, thereby sharpening the appearance ofthe edge 124. Sharpening is a known process in image processing.

Defective pixel detection is a known process in image processing. Aprocessing module 106 which implements defective pixel detection on thepixel value 122 aims to detect whether the pixel value is representativeof a defective pixel by identifying whether it is significantly andunexpectedly different to nearby pixel values. In order to do this theprocessing module 106 will identify the edge 124, so that it can comparethe pixel value 122 to other nearby pixel values which are expected tobe similar because they relate to the same material in the scene(material A).

Defective pixel correction is a known process in image processing.Defective pixel correction may be implemented in the same, or aseparate, processing module to one which implements defective pixeldetection. If the pixel value 122 has been identified as beingrepresentative of a defective pixel, then a processing module 106 whichimplements defective pixel correction on the pixel value 122 aims todetermine a suitable value to assign to the pixel value to replace thedefective pixel value. For example, an average (e.g. the mean or themedian) of the pixel values within the kernel 120 which represent thesame material (material A) could be used to replace the defective pixelvalue 122. In order to do this the processing module 106 will identifythe edge 124 so that it knows which pixel values within the kernel 120are representative of the same material (material A) as is supposed tobe represented by pixel value 122.

De-mosaicing is a known process in image processing. A processing module106 which implements de-mosaicing aims to reconstruct a full colourimage from incomplete colour samples which are output from the imagesensor 102. Interpolation may be used for de-mosaicing, but similar todenoising, the interpolation is only performed over pixel values whichare determined to be representing the same material. So in order toperform de-mosaicing for the pixel value 122, a processing module 106will identify the edge 124 so that it knows which pixel values withinthe kernel 120 are representative of the same material (material A) andcan therefore be used in the interpolation of the de-mosaicing process.

Typically, edge detection methods for detecting edges within a kernelcan only detect a single edge within the kernel.

SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter.

There are a number of problems associated with processing a stream ofpixel values in a camera pipeline using a sequence of processingmodules, such as described in the background section above and shown inFIG. 1 a.

For example, since each of the processing modules 106 in the imageprocessing system 104 operates independently, the different processingmodules 106 may be designed separately. It is possible that inconsistentedge determinations may be made in the different processing modules.Inconsistent edge determinations across the different processing modulescan result in artefacts appearing in the image.

As mentioned above, standard methods for detecting edges within a kernelcan only detect a single edge within the kernel. There may be situationsin which a kernel includes more than one edge, and standard edgedetection algorithms might not be able to detect all of the edges withinthe kernel. An alternative approach is to use a clustering approach inwhich data values within a kernel are grouped (or “clustered”) into oneor more groups. Clustering algorithms are able to group the data valueswithin a kernel into more than two groups, and can cope with cornerswithin the kernel. However, clustering is a problem which is known to be“NP hard”, i.e. non-deterministic polynomial-time hard. As such it isdifficult to implement a clustering algorithm multiple times in multipleprocessing modules for each pixel value of an image in a camera pipelinein real-time as images are captured by the image sensor 102. In fact, itis so difficult that a clustering algorithm has not been used in thisway in the prior art. As an example, current state of the art camerastend to be configured to capture images with many millions of pixels.

Furthermore, different processing modules might be implementingfunctions with at least partially contradictory aims. For example, adenoising function will tend to reduce differences between pixel values,whereas a sharpening function will tend to increase differences betweenpixel values. If each block is implemented separately it can bedifficult to achieve a suitable balance between the contradictoryprocesses, which may tend to fight against each other. Therefore, tuningof the image processing system 104 can be difficult because changes toone processing module 106 may have unforeseen consequences on theoperation of another processing module 106.

According to examples described herein, there is provided an imageprocessing system configured to process a stream of pixel values, theimage processing system comprising:

-   -   a processing module configured to implement processing of a        particular pixel value of the stream by operating on a        particular subset of pixel values of the stream, by:        -   classifying each of the pixel values within the particular            subset into a group of a set of groups;        -   processing the particular pixel value using one or more of            the pixel values of the particular subset in dependence on            the classification of the pixel values of the particular            subset into the groups, wherein said processing the            particular pixel value comprises performing, in a            consolidated operation, multiple processing functions which            depend upon the classification of pixel values of the            particular subset into the groups; and        -   outputting the processed particular pixel value.

The multiple processing functions may comprise two or more of: (i)defective pixel detection, (ii) defective pixel correction, (iii)denoising, (iv) sharpening, and (v) de-mosaicing. For example, the imageprocessing system may be implemented in a camera pipeline.

The image processing system may further comprising a store configured tostore group indication data which indicates one or more groups intowhich pixel values can be grouped, wherein the processing module may beconfigured to process, in each of a plurality of iterations, arespective particular pixel value of the stream by operating on arespective particular subset of pixel values of the stream, by, in eachof the plurality of iterations: retrieving, from the store, groupindication data for at least one group; using the retrieved groupindication data to define the set of groups into which pixel valueswithin the particular subset can be grouped; performing said: (i)classifying each of the pixel values within the particular subset into agroup of a set of groups, and (ii) processing the particular pixel valueusing one or more of the pixel values of the particular subset independence on the classification of the pixel values of the particularsubset into the groups; and storing, in the store, group indication datafor at least one group of the set of groups, for use in a subsequentiteration.

Said storing group indication data for at least one group may comprisestoring, as an indicative value for a group, an average value of thepixel values which are classified as being part of that group. Saidstoring group indication data for at least one group may comprisestoring at least one of: (i) an indication of the spread of the datavalues which are classified as being part of that group, (ii) anindicative position of the data values which are classified as beingpart of that group, and (iii) an indication of the number of members ofthe group.

Each of the pixel values within the particular subset may be classifiedinto one of the groups of the set of groups based on the value of thatpixel value.

The image processing system may be configured to detect an edge withinthe particular subset, wherein each of the pixel values within theparticular subset is classified into one of the groups of the set ofgroups based on the relative position of the pixel value compared to thedetected edge.

Group indication data for a group may indicate a range of pixel valueswhich are to be classified into the group. The group indication data fora group may comprise an indicative value within the range of pixelvalues which are to be classified into the group. The extent of therange of pixel values around the indicative value may be defined by acalibration process. The calibration process may be a static calibrationprocess, a dynamic calibration process or a real-time calibrationprocess.

The group indication data for a group may comprise indications of theupper and lower bounds of the range of pixel values which are to beclassified into the group.

The stream of pixel values may represent an image as a two-dimensionalarray of pixel values, wherein each of the subsets of pixel valueswithin the stream of data values may represent a contiguous block ofpixel values within the two-dimensional array. A particular subset ofpixel values may represent a block of pixel values which includes andsurrounds the corresponding particular pixel value.

In some examples, the set of groups comprises up to n groups, and theprocessing module may be further configured to: after the classificationof the pixel values of the particular subset into the groups, and priorto said processing the particular pixel value, selectively discard oneor more of the groups based on the number of pixel values which areclassified into each of the groups, such that up to m groups aremaintained, where n≥m. Said processing the particular pixel value usingone or more of the pixel values of the particular subset in dependenceon the classification of the pixel values of the particular subset intothe groups may comprise: if the particular pixel value is within amaintained group, determining that the particular pixel value is notrepresentative of a defective pixel; and if the particular pixel valueis not within a maintained group, determining that the particular pixelvalue is representative of a defective pixel.

The processing module may be configured to: determine, for each of atleast one group of the set of groups, a centroid representative of aposition of the group; and determine, for each of the at least one groupof the set of groups, an average value of the pixel values within thegroup. Said processing the particular pixel value may comprise applyingone or more of denoising, sharpening and de-mosaicing to the particularpixel value based on the at least one determined centroid of the atleast one group. Said processing the particular pixel value maycomprise, if the particular pixel value is determined to berepresentative of a defective pixel: selecting one or more of themaintained groups based on the determined centroids of the groups, andreplacing the particular pixel value with a value based on thedetermined one or more average values for the one or more selectedgroups.

There is provided a method of processing a stream of pixel values in animage processing system, the method comprising:

-   -   implementing processing of a particular pixel value of the        stream by operating on a particular subset of pixel values of        the stream, by:        -   classifying each of the pixel values within the particular            subset into a group of a set of groups;        -   processing the particular pixel value using one or more of            the pixel values of the particular subset in dependence on            the classification of the pixel values of the particular            subset into the groups, wherein said processing the            particular pixel value comprises performing, in a            consolidated operation, multiple processing functions which            depend upon the classification of pixel values of the            particular subset into the groups; and        -   outputting the processed particular pixel value.

In some examples, there is provided a data processing system configuredto process a stream of data values, the data processing systemcomprising:

-   -   a store configured to store group indication data which        indicates one or more groups into which data values can be        grouped; and    -   a processing module configured to process, in each of a        plurality of iterations, a respective particular data value of        the stream by operating on a respective particular subset of        data values of the stream, by, in each of the plurality of        iterations:        -   retrieving, from the store, group indication data for at            least one group;        -   using the retrieved group indication data to define a set of            groups into which data values within the particular subset            can be grouped;        -   classifying each of the data values within the particular            subset into one of the groups of the set of groups;        -   processing the particular data value using one or more of            the data values of the particular subset in dependence on            the classification of the data values of the particular            subset into the groups; and        -   storing, in the store, group indication data for at least            one group of the set of groups, for use in a subsequent            iteration;    -   wherein the processing module is configured to output the        processed data values.

There is provided an image processing system configured to process pixelvalues, the image processing system comprising a processing moduleconfigured to:

-   -   receive a plurality of pixel values; and    -   implement processing of a particular pixel value by operating on        a particular subset of the received pixel values, by:        -   defining a set of one or more groups into which pixel values            within the particular subset can be grouped;        -   classifying each of the pixel values within the particular            subset into one of the groups of the set of one or more            groups based on the value of that pixel value;        -   processing the particular pixel value using one or more of            the pixel values of the particular subset in dependence on            the classification of the pixel values of the particular            subset into the one or more groups; and        -   outputting the processed particular pixel value.

There is provided an image processing system configured to process pixelvalues, the image processing system comprising a processing moduleconfigured to:

-   -   receive a plurality of pixel values, each of the received pixel        values having a first number of bits; and    -   implement processing of a particular pixel value by operating on        a particular subset of the received pixel values, by:        -   classifying each of the pixel values within the particular            subset into a group of a set of one or more groups;        -   determining an average value in respect of the pixel values            within the particular subset which are classified into one            of the one or more groups, wherein the determined average            value has a second number of bits, wherein said second            number is greater than said first number;        -   replacing the particular pixel value based on the determined            average value; and        -   outputting the processed particular pixel value.

The data processing systems and image processing systems describedherein may be embodied in hardware on an integrated circuit. There maybe provided a method of manufacturing, at an integrated circuitmanufacturing system, a data processing system and/or an imageprocessing system as described herein. There may be provided anintegrated circuit definition dataset that, when processed in anintegrated circuit manufacturing system, configures the system tomanufacture a data processing system or an image processing system asdescribed herein. There may be provided a non-transitory computerreadable storage medium having stored thereon a computer readabledescription of an integrated circuit that, when processed, causes alayout processing system to generate a circuit layout description usedin an integrated circuit manufacturing system to manufacture a dataprocessing system or an image processing system as described herein.

There may be provided an integrated circuit manufacturing systemcomprising:

-   -   a non-transitory computer readable storage medium having stored        thereon a computer readable integrated circuit description that        describes a data processing system or an image processing system        as described herein;    -   a layout processing system configured to process the integrated        circuit description so as to generate a circuit layout        description of an integrated circuit embodying the data        processing system or the image processing system as described        herein; and    -   an integrated circuit generation system configured to        manufacture the data processing system or the image processing        system according to the circuit layout description.

Although methods described herein are best suited to being implementedin dedicated hardware, the methods could be implemented in softwarerunning on general purpose hardware. Therefore, there may be providedcomputer program code for performing any of the methods describedherein. There may be provided non-transitory computer readable storagemedium having stored thereon computer readable instructions that, whenexecuted at a computer system, cause the computer system to perform anyof the methods described herein.

The above features may be combined as appropriate, as would be apparentto a skilled person, and may be combined with any of the aspects of theexamples described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawings will be provided by the Office upon request and paymentof the necessary fee. Examples will now be described in detail withreference to the accompanying drawings in which:

FIG. 1a shows an example image processing system of the prior art;

FIG. 1b shows a 2D array of pixel values representing an image;

FIG. 2 shows an image processing system according to examples describedherein;

FIG. 3 is a flow chart for a method of processing a stream of datavalues;

FIG. 4 shows a 2D array of data values, highlighting a first subset ofthe data values which will be used for processing a first data value inthe 2D array;

FIG. 5 shows a 2D array of data values, highlighting a second subset ofthe data values which will be used for processing a second data value inthe 2D array;

FIG. 6 illustrates ranges of data values spanned by different groups ofdata values;

FIG. 7 shows a calibration curve used to determine the extent of a rangeof a group as a function of the average value of the group;

FIG. 8 is a flow chart showing some of the steps involved in processinga pixel value in examples described herein;

FIG. 9a shows a representation of a first example image which isreceived at a processing module;

FIG. 9b shows a representation of the first example image when it isoutput from the processing module;

FIG. 10a shows a representation of a second example image which isreceived at the processing module;

FIG. 10b shows a representation of the second example image when it isoutput from the processing module;

FIG. 11 is a flow chart for a method of implementing dynamic rangeenhancement; and

FIG. 12 shows an integrated circuit manufacturing system for generatingan integrated circuit embodying a data processing system.

The accompanying drawings illustrate various examples. The skilledperson will appreciate that the illustrated element boundaries (e.g.,boxes, groups of boxes, or other shapes) in the drawings represent oneexample of the boundaries. It may be that in some examples, one elementmay be designed as multiple elements or that multiple elements may bedesigned as one element. Common reference numerals are used throughoutthe figures, where appropriate, to indicate similar features.

DETAILED DESCRIPTION

The following description is presented by way of example to enable aperson skilled in the art to make and use the invention. The presentinvention is not limited to the embodiments described herein and variousmodifications to the disclosed embodiments will be apparent to thoseskilled in the art.

Embodiments will now be described by way of example only. According toembodiments described herein, a stream of data values (e.g. pixelvalues) is processed in a data processing system (e.g. an imageprocessing system such as a camera pipeline). As an example, processedpixel values may represent an image.

It has been appreciated that where a processing module processes aparticular data value from the stream by operating on a respectivesubset of data values surrounding the particular data value (e.g. akernel of data values centred on the particular data value that iscurrently being processed), the subset of data values which are operatedon for a particular data value in the stream will at least partiallyoverlap with the subset of data values which are operated on for another(e.g. the next) data value in the stream. Therefore, some informationrelating to the subset of data values for processing one data value inthe stream is relevant to, and can be used for, the subset of datavalues for processing the next data value in the stream. In other words,information from processing a current data value in the stream (e.g.group indication data which indicates one or more groups into which datavalues can be grouped) can be used to simplify the task of processingthe next data value in the stream (e.g. for defining a set of groupsinto which data values can be grouped). In this way, the processinginvolved in performing clustering on the data values within a subset ofdata values can be shared between consecutive data values in the stream.This reduces the latency and power consumption of the image processingsystem, in contrast to prior art systems in which each data value in thestream is processed separately.

In particular, the Bayesian principle of using prior knowledge cansimplify the clustering problem. Groups which are determined forclustering of data values within a previous subset (the precedingsubset) are likely to be similar to the groups which are determined forclustering of data values within a current subset because the twosubsets partially overlap with other (e.g. the two subsets maysubstantially overlap with each other). Therefore, we can use theprevious groups defined for the previous data value as a starting pointfor defining the groups to use for the current data value. When thecurrent data value has been processed, group indication data for thegroups that have been used for the current data value can be stored, sothat in the next iteration (i.e. when processing the next data value ofthe stream) the group indication data can be retrieved and used todefine the groups into which data values within the next subset can begrouped. This greatly simplifies the clustering problem so that it isdeterministically solvable in real-time (i.e. it is no longer an NP hardproblem) even on devices which have tight restrictions on processingpower and size (e.g. mobile devices such as handheld cameras, smartphones and tablets, etc.).

Therefore, in some examples described herein, an edge detectionalgorithm is not used to identify groups of data values within a kernel,and instead a clustering algorithm is used to identify the groups ofdata values within the kernel. Edge detection usually results in abinary split of the data values within a kernel into two groups: onegroup on either side of the detected edge. For example, as describedabove, FIG. 1b shows the pixel values being classified as either A or Bbased on their positions relative to the edge 124. In this way, the edge124 is thinned (or thresholded) down to a single pixel width. However,in reality, edges in images are often soft, and may have finite widthssuch that some pixels can lie within an edge rather than on one side orthe other, such that a set of “fuzzy”, or half-toned, pixel values mayexist along the edge. Edge detectors will split these edge pixel valuesinto one of the two groups defined by the edge. Making the wrong choiceor ascribing an intermediate value (lying on or near the edge 124) toeither group A or B (as shown in FIG. 1b ) may lead tomis-classification of pixels, which in turn may lead to poor performanceand the introduction of errors along or near the edge 124 in theprocessed image data. However, in contrast, a clustering algorithm maygenerate a separate group for the intermediate pixel values which arebetween the values of pixels within groups A and B. In particular, aclustering algorithm can easily group the pixel values within a kernelinto more than two groups. Furthermore, in a clustering algorithm, thegrouping of the pixel values within a kernel does not depend upon thespatial position of the pixel values within the kernel. In this sensethe clustering algorithm is free to classify a pixel value that iswithin a kernel to a group without considering the spatial position ofthe pixel value within the kernel. Instead, a set of groups are definedinto which pixel values within a particular subset can be grouped, andthen each of the pixel values within the particular subset areclassified into one of the groups based on the value of that pixelvalue. For example, the groups may be defined by defining a range ofpixel values for each of the groups, wherein each of the pixel valueswithin a particular subset is grouped into one of the groups bycomparing the pixel value to one or more of the defined pixel valueranges for the groups.

The clustering approach used in examples described herein can cope with:(i) any angle of edge (i.e. the angle of an edge is not considered),(ii) corners (e.g. if a quadrant within the kernel is different to therest of the kernel, (iii) more than one edge within the kernel, and (iv)soft edges (e.g. edges with a finite width which may span over one ormore pixels, such that some pixels may be located within an edge). Thisis in contrast to standard edge detectors which cannot cope with any ofthe four situations mentioned above.

Furthermore, since data is shared between consecutive data values in thestream, it is possible to use a smaller kernel size and still retaingood processing results compared to prior art systems in which each datavalue in the stream is processed separately. For example, a 5×5 kernelcould be used rather than a 7×7 kernel, which approximately halves thenumber of data values in the kernel. A reduction in the size of thekernels tends to reduce the silicon area of the image processing systemand reduce the processing power consumed by the image processing system.

Furthermore, in examples described herein, multiple processing functions(which would typically be performed by separate processing modules)which depend upon the classification of pixel values of a subset intogroups, are performed in a consolidated operation (e.g. by a singleprocessing module). This means that the clustering process only needs tobe performed once for the multiple processing functions. The multipleprocessing functions may for example comprise two or more of: (i)defective pixel detection, (ii) defective pixel correction, (iii)denoising, (iv) sharpening, and (v) de-mosaicing. Consolidating multiplefunctions can reduce the number of processing modules and the number ofline store modules in the image processing system, which will reduce thesilicon area and power consumption of the image processing system.Furthermore, by applying the multiple processing functions in aconsolidated operation (e.g. with a single processing module) it issimpler to tune the operation of the functions compared to when thedifferent functions are implemented by separate blocks. The tuning isparticularly simplified for functions which may tend to fight againsteach other, such as denoising and sharpening. The tuning is alsosimplified for functions which replicate some functionality, such asdefective pixel detection and defective pixel correction. Furthermore,by performing, in a consolidated operation, multiple processingfunctions which depend upon the classification of data values of theparticular subset into the groups, the classification of the data valuesinto the groups is consistent for the multiple processing functions,thereby avoiding artefacts which may occur due to inconsistentclustering decisions for different functions.

Furthermore, in some examples, a pixel value can be replaced with anaverage of pixel values within a group of similar pixel values. Thisaveraging process can be used to enhance (i.e. increase) the dynamicrange of the pixel values. For example, an average value can bedetermined with more bits than the number of bits in each of theindividual input pixel values.

The processing modules in the image processing system may be implementedin hardware, e.g. in fixed function circuitry. Dedicated fixed functionhardware is typically more efficient at processing data values (in termsof processing speed and power consumption) compared to implementing theprocessing modules by running software on general purpose hardware.Implementing the processing modules in hardware is not as flexible asimplementing the processing modules in software in the sense that theoperation is difficult (or impossible) to change after the hardware hasbeen manufactured, whereas software can easily be changed by loadingdifferent software code into a memory storing the software to beexecuted. However, in a data processing system such as a camera pipelineor other system which processes a stream of data values in a fixedmanner in real-time, the advantages of implementing the processingmodules in hardware (in terms of processing speed and power consumption)tend to outweigh the disadvantage of the inflexibility of a hardwareimplementation. By implementing multiple processing functions in asingle consolidated operation, the number of processing modules isreduced in the image processing system, thereby significantly reducingthe size (in terms of silicon area) of the hardware implementing thedata processing system. For example, four or five distinct processingmodules in a prior art system could be consolidated down to just oneprocessing module in examples described herein.

Examples are described below with reference to an image processingsystem, but concepts described herein could be applied to other types ofdata processing systems (e.g. audio processing systems and signalprocessing systems) which process streams of data values. FIG. 2illustrates an image processing system 204 which is arranged to receiveimage data (e.g. pixel values) from an image sensor 202. The image datareceived from the image sensor 202 may be referred to as “raw” imagedata or “unprocessed” image data. As described above, an image sensor,such as the image sensor 202, can include an array of sensors (e.g.charge-coupled devices (CCDs) or complementary metal-oxide-semiconductor(CMOS) sensors) which can sense incident light at respective pixelpositions to determine raw image data (e.g. unprocessed pixel values)which can be fed into the image processing system 204. The imageprocessing system 204 could be described as a camera pipeline. Thesystem shown in FIG. 2 may be implemented in a device, such as a camera.In some examples, the image sensor 202 may capture a sequence of images,which can be used to represent frames of a video sequence. The imagedata may be in one of many different possible formats, for example thepixel values may be monochrome pixel values, wherein each pixel valuerelates to the same, single colour channel. In other examples, the pixelvalues may comprise Red, Green and Blue (RGB) values to represent theintensity of the respective three colour channels at the respectivepixel positions. In some examples, the raw image data may relate tomultiple colour channels, but the raw image data for each particularpixel position may include only a single colour value (e.g. a red, greenor blue colour value), arranged in a pattern such as a Bayer pattern,and in these examples a de-mosaicing process may be performed such thata full set of colour values (e.g. three colour values, such as red,green and blue colour values) are determined for each pixel position. Ade-mosaicing process is described below. Other colour formats may beused in other examples. The image sensor 202 may provide the pixelvalues to the image processing system 204 in a raster scan order, suchthat pixel values for lines of pixels (e.g. rows of pixels) are receivedand processed in the image processing system 204 as they are receivedfrom the image sensor 202 in real-time.

The image processing system 204 may comprise a number of processingmodules 206, 210 and 214 for applying respective processing functions tothe image data. The processing modules may, for example, be implementedin dedicated hardware modules (e.g. in fixed function circuitry) suchthat they can be optimised for performing specific processing functionson the pixel values in an efficient manner (e.g. “efficient” in terms ofat least one of latency, power consumption and silicon area). The imageprocessing system 204 also comprises a store 212 which is coupled to theprocessing module 210. As described in more detail below, the store 212is configured to store group indication data which indicates one or moregroups into which pixel values can be grouped. The processing module(s)206 and 214 are represented with dashed boxes because the number ofthese modules may vary in different implementations and that number maybe zero in some implementations. The processing module(s) 206 and 214may perform any suitable functions on pixel values, e.g. adjusting gainsof the pixel values, adjusting the white balance of the pixel values,etc. If a processing module requires access to pixel values from morethan one line of the image (e.g. a 2D filtering module) then a linestore module (or “line store bank”) can be implemented in the imageprocessing system 204 prior to the processing module. For example,processing module 210 requires access to multiple rows of pixel values,e.g. the processing module 210 may be configured to implement a 2Dfiltering process, defective pixel detection, defective pixelcorrection, sharpening and/or de-mosaicing. As described above, in thesetypes of processes, a particular pixel value can be processed based onthe pixel values within a kernel of pixel values including theparticular pixel value (e.g. centred on the particular pixel value). Theline store module 208 is implemented to store pixel values for multiplelines of pixels and can provide pixel values from multiple lines to theprocessing module 210. Processed image data is output from the imageprocessing system 204, and may be used in any suitable manner, e.g.output to a display, stored in memory, transmitted to another device,etc.

Operation of the image processing system 204 is described with referenceto the flow chart shown in FIG. 3. The image sensor 202 captures imagedata, e.g. pixel intensity values for different pixels within an image.Techniques for capturing these pixel intensity values at the imagesensors are known in the art and the details of this process are beyondthe scope of this description. In step S302 the image processing system204 receives a stream of data values from the image sensor 202. Thepixel values are fed from the image sensor 202 to the image processingsystem 204 in a particular order, thereby forming a stream of pixelvalues, e.g. raster scan order in which rows of pixel values arereceived in turn at the image processing system 204. It would bepossible for the pixel values to be passed to the image processingsystem 204 in a different order, e.g. Boustrophedon order, and it ispossible for multiple lines of pixel values to be passed to the imageprocessing system 204 at once, in parallel. However, since the imagesensor 202 and the image processing system 204 may be designed bydifferent parties, it is convenient for there to be a presumed, orstandard, order in which pixel values are passed from the image sensor202 to the image processing system 204, and raster scan order is theusual order used. Therefore, in the examples described herein we referto receiving rows of pixel values in a raster scan order, but it shouldbe apparent that other orders could be used, and that columns of pixelvalues could be received rather than rows, so where we refer to “rows”of pixel values it is to be understood that we are generally referringto “lines” of pixels values, which could for example be rows or columns.

As described above, the image processing system 204 may, or may not,perform some processing on the pixel values in the processing module(s)206 before the pixel values are stored in the line store module 208. Asdescribed above, the line store module 208 allows pixel values frommultiple rows to be provided to the processing module 210 together.

FIG. 4 shows an example in which the stream of pixel values represents atwo-dimensional array of pixel values 402 forming an image, wherein thearray includes six rows of pixel values, with each row including sevenpixel values. Obviously, in more complex examples, there may be manymore pixel values within an image, e.g. millions of pixel valuesarranged such that there are hundreds or thousands of rows and hundredsor thousands of columns of pixels.

The processing module 210 processes a particular pixel value 404 byoperating on the pixel values within the 5×5 kernel 406, which surroundsand includes pixel value 404 (e.g. the kernel 406 is centred on pixelvalue 404). In step S304, the processing module 210 receives the twentyfive pixel values in the kernel 406 from the line store module 208. Thekernel 406 defines a subset of pixel values which represents acontiguous block of pixel values within the two-dimensional array ofpixel values representing the image.

In step S305 a subset index subset index i is set such that i=0, for theprocessing of a first subset (e.g. the kernel 406).

In step S306, the processing module 210 defines a set of one or moregroups into which pixel values of a first subset (i.e. pixel valueswithin the kernel 406) can be grouped. This can be done by analysing thepixel values of the subset 406 and performing a clustering process togroup the pixel values into groups based on their pixel values. This mayinvolve an edge detection process to identify edges between differentmaterials (i.e. sharp changes in appearance) in the image. FIG. 4 showstwo edges (408 and 410) in the scene, such that three distinct areas areformed within the kernel 406. Therefore, three groups are defined (A, Band C).

In step S308 the processing module 210 classifies each of the pixelvalues of the subset 406 into one of the groups (A, B or C) of the setof groups. In FIG. 4 each pixel value is denoted with either “A”, “B” or“C” to represent which group the pixel value has been classified into.

In step S310 the processing module 210 processes the pixel value 404 independence on the classification of the pixel values of the subset 406into the groups (A, B or C). The processing performed by the processingmodule 210 may, for example, involve one or more of defective pixeldetection, defective pixel correction, denoising, sharpening andde-mosaicing. It is noted that the denoising (or filtering) referred toherein is performed in the spatial domain, i.e. it is spatial denoising(or spatial filtering). If the processing module 210 performs more thanone processing function, these processing functions may be performed ina consolidated manner, such that contradictory aims of the differentprocessing functions can be controlled in a consistent manner, and suchthat tuning of the image processing system is simplified.

In step S312 the processed pixel value 404 is outputted from theprocessing module 210 for use in the image processing system 204. Theprocessed pixel value may be used by other processing modules 214 withinthe image processing system 204, and/or outputted from the imageprocessing system 204.

In step S314 the processing module 210 stores group indication data inthe store 212, which can be used when processing the next pixel value.The group indication data provides an indication of the three groups (A,B and C) which were used when processing the pixel value 404. Forexample, step S314 may involve storing, as an indicative value for agroup, an average value μ (e.g. the mean value or the median value) ofthe pixel values which were classified as being part of that group instep S308. Therefore, as an example, three indicative values μ_(A),μ_(B) and μ_(C) may be stored to indicate the three groups A, B and C.As described in more detail below, the group indication data which isstored for a group may include a centroid position for the group, whichmay be used as part of a defective pixel correction process to choose asuitable group from which an average value can be used to replace adefective pixel value.

FIG. 5 shows a two dimensional array of pixel values 502 forming animage, which is the same image as that described above with reference toFIG. 4. FIG. 5 illustrates the situation when the processing module 210is processing the next pixel value 504 after the pixel value 404 shownin FIG. 4 (where “next” refers to a raster scan order in this example).In order to process the pixel value 504, the processing module 210operates on a kernel 506 which defines a subset of pixel values in theimage. Since the image is the same in FIGS. 4 and 5, the edges 408 and410 are in the same place in both Figures. It can be appreciated thatthe subset of pixel values 506 partially overlaps with the subset ofpixel values 406 which were used to process the pixel value 404. In stepS316, the index i is incremented, such that on the first iteration stepS316 involves setting the subset index to 1.

In step S318 the processing module 210 retrieves the group indicationdata stored for the previous subset (i.e. the (i−1)^(th) subset) fromthe store 212. The group indication data indicates, for a group, therange of values which fall into that group. The group indication datamay take different forms in different examples, but as described abovethe group indication data for a group may represent an indicative valuefor the group (e.g. determined as an average (e.g. mean or median) valueof the pixel values which were included in that group in the previousiteration). As an example, three indicative values (μ_(A), μ_(B) andμ_(C)) may be retrieved in step S318, which have values represented inFIG. 6. FIG. 6 illustrates a scale 600 of possible pixel values, rangingfrom 0 to the maximum possible pixel value (“Max”) which is, forexample, set by the number of bits in a pixel value. For example, if apixel value has eight bits, then the maximum integer pixel value whichcan be represented is 255, so the full range of possible pixel valueswould span from 0 to 255. It is clear that, in other examples, othernumbers of bits could be used for the pixel values such that the valueof Max can be different. Furthermore, in some examples, pixel values maybe in a format in which they are allowed to be negative such that zerowould not be at the bottom of the range. The group indication data for agroup which is retrieved in step S318 may include an indication of acentroid position for the group.

In the example described above, a pixel value comprises a single colourcomponent. For example, in a monochrome image, each pixel value relatesto the same, single colour channel. For simplicity and clarity, anexample is described herein in relation to the pixel values of amonochrome image. However, as explained in more detail below, in anotherexample, an image may include pixel values which collectively relate tomultiple colour channels, but it may the case that each particular pixelvalue relates to a single one of those colour channels. For example, animage sensor may generate pixel values in accordance with apredetermined pattern (e.g. a Bayer pattern, such that some pixel valuesrepresent Red colour values, some pixel values represent Blue colourvalues, and some pixel values represent Green colour values). Ade-mosaicing process can be applied to generate full colour data (e.g.three colour components for each pixel). In a further example, eachpixel value may comprise a plurality of colour components (e.g. after ade-mosaicing process has been performed). As an example, a pixel valuecould comprise three components (e.g. red, green and blue components),each of which may for example have eight bits. The concepts used in themethods described herein, which are primarily described with referenceto a monochrome image comprising pixel values relating to a singlecolour channel, could be used for processing chromatic images in whichpixel values comprise data relating to a plurality of colour channels(either before or after de-mosaicing). It is noted that, in examplesdescribed herein, where a pixel value has multiple components theclassification of the pixel into one of the groups is determined onceper pixel (not once per component), i.e. each component of the pixelvalue is classified into the same group. This ensures that consistentclustering decisions are made across all the colour channels.

In step S320, the processing module 210 uses the retrieved groupindication data (e.g. the indicative values μ_(A), μB and μ_(C)) todefine a set of groups into which data values within the i^(th) subset506 can be grouped. For example, a calibration process can be used todefine an extent of a range of data values around an indicative value μfor a group. For example, FIG. 7 is a graph showing a calibration curve702. The calibration curve shows a typical standard deviation that isexperienced due to noise in the image sensor 202, as a function of thepixel value. As the pixel value increases the standard deviation tendsto increase. The calibration curve 702 can be pre-set (e.g. duringdesign time) based on a likely noise experienced by a generic imagesensor, such that the calibration curve is not specifically tailored forthe particular image sensor 202. As an alternative, the calibrationprocess may be performed on the sensor at manufacture time (e.g. when ithas been manufactured) and the calibration curve 702 may be stored atthe manufacture time (this may be referred to as a “static” calibrationprocess). Alternatively, the calibration curve 702 could be set bytesting the specific image sensor 202 to detect a standard deviation fordifferent pixel values. In other words, the sensor noise curve (or“calibration curve”) may be generated from an analysis of the noise in areal-time image. This could be done by using the image sensor 202 todetermine pixel values for an object with a uniform colour andbrightness, and determining differences in the pixel values. By alteringthe colour and brightness of the object, typical standard deviationscould be detected across the range of pixel values, thereby determiningthe calibration curve 702. For example, the calibration of the sensorcould be performed based on earlier, recently processed images (this maybe referred to as a “dynamic” calibration process). Some examples mayuse a “real-time” calibration process in which the calibration isdetermined for an iteration and results of the calibration are storedwith the group indication data, such that they can be used on the nextiteration.

The calibration curve 702 can be used to define ranges of pixel valuesaround the indicative values μ_(A), μ_(B) and μ_(C), such that threegroups 602 _(A), 602 _(B) and 602 _(C) can be defined. The extents ofthe ranges of the groups 602 _(A), 602 _(B) and 602 _(C) are shown withdashed lines in FIG. 6. The extents of the ranges may be determined indifferent ways in different examples. To give some examples, a rangearound an indicative value μ may be given by ±σ, ±2σ or ±σ², where σ isthe standard deviation as indicated by the calibration curve 702 for theindicative value μ (such that σ² is the variance). It can be seen thatthe size of the value range 602 _(A) is smaller than the size of thevalue range 602 _(B), which itself is smaller than the size of the valuerange 602 _(C). This is because, according to the calibration curve 702,the range of a group extends further from the indicative value μ forlarger indicative values μ. There may be a respective calibration curvefor each colour component (e.g. red, green and blue components), orthere may be a single calibration curve which is used for all of thecolour components. In the example described above the extent of a rangedepends only on the expected noise (e.g. as indicated by the calibrationcurve). However, in some examples the variation may be dependent uponfactors (e.g. surface shading or texture) other than the noise.Furthermore, in some examples, a calibration curve is not used to definethe ranges. For example, the extents for each range could be of equalsize, which may be fixed or variable.

In the example given above, the indicative value μ for a group is theaverage value of the data values in that group, such that the grouprange extends either side of the indicative value μ. In other examples,the indicative value could represent another value within the range ofvalues of a group, e.g. the indicative value for a group could be theminimum or maximum of the values within a group such that the grouprange extends on only one side of the indicative value (e.g. only abovethe indicative value if the indicative value is a minimum value of agroup, and only below the indicative value if the indicative value is amaximum value of a group).

Furthermore, although in the example described above, the groupindication data that is stored for a group in the store 212 is just theindicative value μ, (e.g. where the extents of the group around theindicative value are inferred from the calibration curve 702), in otherexamples it would be possible to store, as the group indication data fora group, the upper and lower bounds of the range of pixel values whichare to be classified into the group. This would mean that more data(e.g. twice as much data in a naïve system) would be stored in the store212 to indicate the groups, but there would be no need to use acalibration curve. In both of these examples, the group indication datafor a group indicates (perhaps indirectly) a range of data values whichare to be classified into the group.

Three groups have been defined which correspond with the groups 602_(A), 602 _(B), 602 _(C) that were used in the previous iteration.Further groups are defined by the gaps between these groups, such thatup to seven groups are determined in this example, as indicated by theranges 604 ₁ to 604 ₇. Therefore, in this example, the processing module210 has defined seven groups 604 which collectively span the entirerange of possible data values. The groups 604 are contiguous so they donot overlap with each other. This means that for every possible pixelvalue (from zero to Max in the example shown in FIG. 6) one, and onlyone, group 604 is defined. In some cases, fewer than seven groups may bedefined, e.g. if group 602 _(A) started at zero.

It is noted that in this 1D example, where the data values aresingle-component values, the groups 602 _(A), 602 _(B) and 602 _(C) aresimple to define, and the gaps between those groups create furtherwell-defined groups. In other examples in which the data values havemultiple components (e.g. Red, Green and Blue components), groups can bedefined as regions (e.g. cuboids, or other simple shapes) surroundingthe indicative values (μ) for the groups. The regions are chosen so thatthey do not overlap with each other. In these multi-component examples,the group/region definitions can be updated in much the same way as thegroups in the 1D example described in detail herein. However, the gapsbetween regions are not so well defined. One option for handling thegaps in a multi-component example is to not define groups for the gaps.This would mean that some possible pixel values would not fall into oneof the groups. Another option for handling the gaps in a multi-componentexample is to define a single “gap” group, whereby any pixel value whichdoes not fall into one of the regions (e.g. cuboids) surrounding anindicative value for a group is categorised into the “gap group”. Inthis way “good space” can be tracked by the regions (e.g. cuboids) andthe rest of the value space can be flagged as “bad” (or “noise”,“broken”, “needs fixing”).

In general, group indication data for up to m groups may be retrievedfrom the store 212 in step S318, wherein the defined set of groups intowhich pixel values can be grouped comprises up to n groups, where n≥m,with the extra defined groups being defined by the gaps between andaround the retrieved groups. In general m≤n≤2m+1, and as an example nmay be equal to 2m+1. In the 1D example given above in which there is agap either side of each of the retrieved groups, m=3 and n=7.

In step S322 the processing module 210 classifies each of the pixelvalues within the i^(th) subset 506 into one of the groups 604 definedin step S320. Each of the pixel values within the subset 506 isclassified into one of the groups 604 based on the value of that pixelvalue. In other words, if the pixel value falls into the value range fora particular group then the pixel value is classified into thatparticular group. In FIG. 5 each pixel value is denoted with either “A”,“B” or “C” to represent which group the pixel value has been classifiedinto. Group A corresponds to group 604 ₂, group B corresponds to group604 ₄ and group C corresponds to group 604 ₆.

Steps S318 to S322 are much simpler to implement than steps S306 andS308. Steps S318 to S322 turn the clustering problem into one which isnot NP hard, and is solvable in real-time even on devices such ashandheld cameras, smartphone and tablets which have tight restrictionson size and power consumption. This is achieved by applying the Bayesianapproach such that knowledge of the groups used in the previousiteration is used as a starting point (i.e. as “priors”) for definingthe groups to use for the current iteration. Steps S318 and S320 definethe groups 604 without needing to analyse the pixel values in the subset506, and so these steps can be performed very quickly on hardware whichcovers a small silicon area, with little power consumption. Furthermore,once the groups (604 ₁ to 604 ₇) have been defined, it is simple toclassify each of the pixel values into one of the groups by comparingthe pixel value to the ranges of the values covered by the groups. Theclassification of a pixel value from the subset 506 into one of thegroups does not depend upon the position of the pixel value within thesubset 506. Therefore the classification process does not need toanalyse the positions of the pixel values, and can simply compare thepixel value to the ranges of values defined for each of the groups.Since the classification of a pixel value does not depend upon theposition of the pixel value and does not look for specific edges, it canbe a simpler process than an edge detection process which attempts toidentify the positions of the edges 408 and 410 within the image inorder to determine regions within the kernel 506 which are associatedwith the different materials in the scene, and can then classify eachpixel value based on its position and the positions of the determinedregions. One other advantage to the grouping method of steps S318 toS322 is that the orientation of edges in the image does not affect theeffectiveness of the grouping process. This is in contrast to some knownedge detection algorithms such as a Sobel filter which only looks foredges at certain angles. For example, a Canny filter typically uses avertical Sobel filter and a horizontal Sobel filter, and then attemptsto combine them into a general filter; whereas by contrast, theclustering approach taken in the examples described herein can cope withall angles and curved edges and corners, etc.

As described in more detail below, after the pixel values in the subset506 have been classified into the seven groups 604 ₁ to 604 ₇, and priorto processing the pixel value 504 in step S324 some of the groups may bediscarded based on the number of data values which are classified intoeach of the groups 604. With reference to the example described above inwhich group indication data is retrieved in step S318 for up tom groupsand in which up to n groups are defined in step S320, where n>m, thensome of the groups may be discarded, such that m groups are maintained.For example, if m groups are retrieved and n groups are defined then(n−m) groups may be discarded. For example, after step S322, thepopulation of the seven groups 604 ₁ to 604 ₇ can be analysed todetermine which three of the groups 604 are the most highly populated,and those three groups will be maintained (e.g. groups 604 ₂, 604 ₄ and604 ₆ may be maintained), whilst the other four groups will be discarded(e.g. groups 604 ₁, 604 ₃, 604 ₅ and 604 ₇ may be discarded).

In some examples, different numbers of groups are discarded, and in someother examples no groups are discarded at this stage.

It is noted that in the example described above, the group whichincludes pixel 504 itself may have been discarded. However, in someexamples (e.g. examples in which defective pixel correction is not beingimplemented), the group which includes pixel 504 is considered to be aninteresting group (when processing pixel 504 itself), worthy ofmaintaining, even if it is not one of the most highly populated groups.So in these examples, the group including pixel 504 is maintainedirrespective of the population of that group, and then one or more othergroups may also be maintained based on their respective populations.

Furthermore, in some examples, it is possible to merge two or moregroups together to form a merged group. For example, rather thandiscarding a group, the group may be merged with another nearby group(“nearby” in terms of the values represented by the group). For example,two small groups with very similar values may sometimes be betterrepresented as one group.

In step S324 the processing module 210 processes the pixel value 504 independence on the classification of the pixel values of the subset 506into the groups. The processing performed by the processing module 210may for example involve one or more of: (i) defective pixel detectionand/or defective pixel correction), (ii) denoising, (iii) sharpening and(iv) de-mosaicing. For example, the processing performed by theprocessing module 210 may for example involve two or more of: (i)defective pixel processing (e.g. defective pixel detection and/ordefective pixel correction), (ii) denoising, (iii) sharpening and (iv)de-mosaicing. In some examples, the processing module performs aplurality of processing functions including denoising and sharpening. Asdescribed in more detail below with reference to the example flow chartshown in FIG. 8, the processing module 210 may perform more than oneprocessing function in a consolidated operation. This can help tocontrol contradictory aims of the different processing functions in aconsistent manner which can be easily tuned.

In step S326 the processed pixel value 504 is outputted from theprocessing module 210 for use in the image processing system 204. Theprocessed pixel value may be used by other processing modules 214 withinthe image processing system 204, and/or outputted from the imageprocessing system 204.

In step S328 the processing module 210 stores group indication data inthe store 212, which can be used when processing the next pixel value.The group indication data provides an indication of the maintainedgroups (e.g. groups 604 ₂, 604 ₄ and 604 ₆) which were used whenprocessing the pixel value 504 in step S324. For example, step S328 mayinvolve storing, as an indicative value for a group, an average value μ(e.g. the mean value, the median value, a weighted mean value or a meanof a trimmed set of the pixel values in the group (e.g. the mean of theinterquartile of pixel values in a group) to give just some examples ofaverage values which could be used) of the pixel values which wereclassified as being part of that group in step S322. Therefore, as anexample, three indicative values μ_(A), μ_(B) and μ_(C) may be stored toindicate the three groups 604 ₂, 604 ₄ and 604 ₆). The group indicationdata which is stored for a group may include an indicative position(e.g. a centroid position) for the group, which may be used as part of adefective pixel correction process to choose a suitable group from whichan average value can be used to replace a defective pixel value. Otherexamples of indicative positions include a weighted centroid positionand an average trimmed centroid for the group. Furthermore, groupindication data which is stored for a group may include an indication ofthe spread of the data values which are classified as being part of thatgroup, which may be used to define a range for the group. Usefulindications of the spread of the data values include a standarddeviation, a range, and an interquartile range to give just someexamples.

Furthermore, group indication data which is stored for a group mayinclude an indication of relevance of the group (e.g. the number ofmembers of the group, i.e. the number of pixel values which have beenincluded in that group).

In some examples, the pixel values may be grouped into fewer than threedistinct groups (e.g. if all of the pixel values within a subset havethe same pixel value then only one group is populated). In thesesituations, group indication data for fewer than three (i.e. fewer thanm) groups is stored in step S328. This is why in some cases, groupindication data for fewer than three (i.e. fewer than m) groups may beretrieved in step S318 on some iterations.

In step S330 the processing module 210 determines whether there are anymore pixel values to process, and if there are, then the method passesback to step S316 in which the index i is incremented and the methodrepeats by performing steps S318 to S328 for the next pixel value. If itis determined in step S330 that there are no more pixel values toprocess then the method ends in step S332.

In the method shown in FIG. 3 each processed pixel value is outputted asit has been determined (e.g. in steps S312 and S326). It would bepossible to output groups of processed pixel values in batches when awhole batch of pixel values has been processed. For example, when a rowof pixel values have been determined then all of the processed pixelvalues in that row could be outputted together. Furthermore, the methodcould wait until all of the pixel values within an image have beenprocessed before any of them are outputted.

FIG. 8 is a flow chart of an example showing steps that are performed bythe processing module 210 to perform multiple processing functions in aconsolidated operation in step S324. The multiple processing functionscomprise two or more of: defective pixel detection, defective pixelcorrection, spatial denoising, sharpening and de-mosaicing.

Following the classification step S322, the method passes to step S802in which, as described above, one or more of the groups that weredefined in step S320 are selectively discarded, such that m groups aremaintained. As described in the example given above, the groups with thelowest numbers of pixel values are discarded. Therefore, the leastpopulated groups are discarded. Following step S802 m groups aremaintained and the method passes to step S324, which comprises the stepsS804 to S818 as described in more detail below.

In step S804 the processing module 210 determines a centroid for each ofthe maintained groups. The centroid for a group is representative of aposition of the group. This position is a spatial position of a groupwithin the image. A centroid may be determined for a group by finding anaverage (e.g. mean) position of the pixel values which have beenclassified into the group. Alternatively, a centroid may be determinedfor a group by finding a weighted average (e.g. a weighted mean)position for the pixel values which have been classified into the group,e.g. in which higher weights are applied to pixel values which arecloser to the centre of the group compared to the weights applied topixel values which are further from the centre of the group (e.g. wherethe distance may be measured as a Euclidean distance or a Manhattandistance).

In step S806 the processing module 210 determines an average value foreach of the maintained groups. The average value for a group may be amean, or a median, of the pixel values which have been classified intothat group. The average value for a group provides a representativevalue of the pixel values within the group.

In step S808 the processing module 210 determines whether the currentpixel value being processed (e.g. pixel value 504) is within one of themaintained groups. The current pixel value is the i^(th) pixel value,and may be referred to as the particular pixel value.

In the example shown in FIG. 8, defective pixel correction is one of theprocesses being performed. If the current pixel value is not within oneof the maintained groups then the method passes to step S810 in which itis determined that the current pixel value is representative of adefective pixel. For the i^(th) pixel value 504 to not be within one ofthe maintained groups, it must be dissimilar to most of the other pixelvalues in the subset 506, which is interpreted as meaning that the pixelvalue 504 is a defective pixel. In other words, if the pixel value 504did not represent a defective pixel it is unlikely that the pixel valuewould not be in one of the maintained groups. In this way, the defectivepixel detection function has been performed.

In step S812 the processing module 210 selects one or more of themaintained groups, based on the centroids of the maintained groupsdetermined in step S804. For example, the processing module 210 mayselect the closest of the maintained groups to the pixel value 504,based on the centroids of the maintained groups determined in step S804.For example, with reference to FIG. 5, if pixel value 504 was determinedto be representative of a defective pixel (i.e. it was not classified asbeing part of group A) then the centroids of the groups A, B and C wouldbe considered to determine which was closest to the position of thepixel value 504 in the image. In this example, the centroid for group Awould be the closest centroid to the position of the pixel value 504. Inthis example, the centroids for groups B and C would be further than thecentroid for group A from the position of the pixel value 504.Therefore, in this example, group A would be selected in step S812 forthe pixel value 504 (if the pixel value was determined to berepresentative of a defective pixel).

In step S814, the defective pixel is replaced with a value based on thedetermined average value(s) for the selected group(s). If a single groupwas selected in step S812 then the processing module 210 replaces thei^(th) pixel value 504 with the average value of the selected groupwhich was determined in step S806. In the example shown in FIG. 5, thepixel value 504 could be replaced with an average value (e.g. a mean ora median value) of the pixel values in group A. The average of the pixelvalues which are included in the closest maintained group is a suitablevalue to assign to the pixel value 504 if it is determined to bedefective. If there are multiple (e.g. two) substantially equally closemaintained groups, and the pixel value 504 is not within either of them,then it is probable that the pixel value 504 is on or near an edgebetween the closest of the maintained groups. In this scenario, multiplegroups are selected in step S812, and in step S814 a suitable value toreplace the pixel value 504 with is a weighted average of the averagesof the closest maintained groups. The weights of the weighted averagemay be determined based on the distances between the position of thepixel value 504 and the centroid positions of the (e.g. two) closestmaintained groups.

In this way, the defective pixel correction function has been performedin steps S812 and S814. Following step S814 the method passes to stepS818 which is described below.

If, in step S808, it is determined that the current pixel value 504 iswithin one of the maintained groups then the method passes to step S816in which it is determined that the current pixel value 504 is notrepresentative of a defective pixel. For the i^(th) pixel value 504 tobe within one of the maintained groups, it must be similar to at leastsome of the other pixel values in the subset 506, and this isinterpreted as meaning that the pixel value 504 is not a defectivepixel. The method then passes to step S818.

In step S818 the processing module 210 applies one or more of denoising,sharpening and de-mosaicing to the i^(th) pixel value based on thedetermined centroids of the maintained groups. The centroids of thegroups can be used to indicate whether the i^(th) pixel is close to anedge (e.g. edge 408 or 410). For example, the i^(th) pixel may belocated at a position (0,0). If the centroid of another group (i.e. agroup that does not include the i^(th) pixel) is close to the position(0,0) then there is an edge near to the i^(th) pixel. What constitutes“close to” in this context is an implementation decision, and may bedifferent in different examples.

If the i^(th) pixel value is not close to an edge then spatial denoisingcan be applied based on the pixel values within the same group as thepixel value 504 to reduce the appearance of random noise in the image,and in this case little or no sharpening may be applied to the pixelvalue 504. The denoising may involve replacing the pixel value 504 withan average of the pixel values within the same group (e.g. group A). Theaverage may, for example, be a mean, a weighted mean and a mean of atrimmed set of pixel values (e.g. a mean of the interquartile range).The pixel values that are received at the processing module 210 may haveb bits, where to give some examples, b may be 8, 10 or 12. These pixelvalues may originate from a sensor, and it may be the case that one ormore of the least significant bits (LSBs), e.g. the two leastsignificant bits, of the pixel values are essentially representingnoise. This noise may be a combination of one or more of: noiseresulting from analogue to digital conversion (ADC noise), thermalnoise, quantisation noise, power plane noise and random Gaussian noise.In accordance with the Central Limit theorem, all of these types ofnoise get reduced in a deterministic fashion by a denoising processwhich takes the average of a group of pixel values. The clusteringapproach described herein enables a pixel value to be replaced with anaverage of similar pixel values from its local area, i.e. an average ofthe group of which the pixel value is a part. This enables the precisionof the pixel values to be increased, i.e. the noise in the pixel valuescan be reduced. In particular, the signal to noise ratio (SNR) of thepixel values can be better (i.e. higher) than the SNR of the pixelvalues output from the sensor. This is achieved without any temporalaveraging, such that it can be achieved in real-time on pixel values ofan image, e.g. in a camera pipeline. For example as described above, oneor more of the LSBs of the incoming pixel values received at theprocessing module 210 may be dominated by noise (e.g. two of the LSBs ofthe input pixel values may be dominated by noise); whereas the pixelvalues output from the processing module may have fewer LSBs which aredominated by noise (e.g. none or one of the LSBs of the output pixelvalues may be dominated by noise).

If the i^(th) pixel value is close to an edge then sharpening may bemore strongly applied to enhance the sharpness of the edge, e.g. suchthat the difference between the two groups either side of the edge isaccentuated. This can be done by biasing the i^(th) pixel value awayfrom the average value of the pixels in the group on the other side ofthe edge. In this case the spatial denoising does not filter over theedge so as to avoid blurring the edge, and instead the spatial denoisingjust filters over pixel values which have been classified into the samegroup as the i^(th) pixel value.

Some de-mosaicing techniques are known in the art which use groups ofsimilar colour values. Having formed the groups of pixels using thenovel methods described herein, those groups can be used in existingalgorithms that use groups, such as de-mosaicing. In order to prevent,or at least reduce, the introduction of chroma artefacts, de-mosaicingis applied based on the pixel values which have been classified into thesame group as the i^(th) pixel value (and not based on the pixel valueswhich have been classified into other groups). As described above, theclassification of the pixel values into the groups is consistent fordifferent colour channels. An example of de-mosaicing is described withreference to a Bayer image in which 50% of the pixel values are for theGreen channel, 25% of the pixel values are for the Red channel, and 25%of the pixel values are for the Blue channel. The outputs of thede-mosaicing process are Red, Green and Blue values for each of thepixels. The Green plane is processed first to determine values for thegreen component of the pixel values of the current group (i.e. the groupincluding the i^(th) pixel) which did not initially have a Green pixelvalue. This can be done by interpolating, predicting or guessing theGreen component values. Therefore, at the end of this first stage, wehave a Green component value for each pixel in the current group beingprocessed. Intensity differences between pixels of the current group(i.e. the group including the i^(th) pixel) in the Green colour channelcan then be used to interpolate in the Red and Blue colour channels inorder to determine the Red and Blue component values for pixels of thecurrent group which did not initially have Red or Blue pixel valuesrespectively. The determination of the pixel values of the Red and Blueplanes in this second pass can be performed with a bi-linear filter toreconstruct the missing 75% of data, based on weightings determined bythe Green plane. Failure to accurately detect/locate an edge (i.e.failure to correctly assign the pixel values to the groups) may generateerroneous pixels/colours because pixel values from other groups wouldthen distort the determined pixel values for the current group.

Following step S818 the method passes to step S326 described above.

By applying multiple processing functions within the same processingmodule 210, consistent decisions are made about how a pixel value is tobe processed and the processing of pixel values is deterministic andsimple to tune. For example, the denoising and sharpening functions areperformed in a consolidated operation so that the functions don't applycontradictory operations to a particular pixel value. Similarly chromaerrors are less likely to be produced from the de-mosaicing process,leading to a lower acceptable level of spatial denoising, and hence ahigher level of image definition.

In the example shown in FIG. 8 defective pixel correction is one of theprocesses being performed. In some other examples, defective pixelfixing is not one of the processes being performed, and as such stepsS808, S810, S812, S814 and S816 are not performed.

FIG. 8 shows an example in which multiple processing functions areapplied within the same processing module 210, as incorporated into themethod shown in FIG. 3 in which the pixel values are classified intogroups according to a clustering approach as described above. However,in alternative examples, the concept of applying multiple processingfunctions within the same processing module in a consolidated manner isnot reliant on the manner in which the pixel values are classified intogroups. Therefore, in some examples, an edge detection process could beused to classify the pixel values within a subset into one or moregroups, and then the multiple processing functions could be applied in aconsolidated manner, wherein the multiple processing functions dependupon the classification of pixel values of the subset into the groups.Therefore the method may comprise detecting an edge within a currentsubset (e.g. using known edge detection techniques), wherein each of thepixel values within the current subset is classified into one of thegroups of the set of groups based on the relative position of the pixelvalue compared to the detected edge.

FIGS. 9a to 10b illustrate examples in which the processing module 210is used to apply defective pixel detection, defective pixel correction,denoising and sharpening to two test images. These examples do notimplement de-mosaicing, but de-mosaicing could be included in otherexamples.

FIG. 9a shows a representation of a first example image 902 which isreceived at the processing module 210. The image 902 has beenartificially generated as a test image so that it has a large number ofdefective pixels (approximately one in every 300 pixels is defective inthe image 902, whereas a real, poor quality sensor might typically haveone in every 2000 pixels being defective), so the test image 902 isworse in terms of defective pixels than a standard image captured by areal image sensor. The image 902 has random noise applied to it, whichcan be seen in the ‘speckled’ nature of the otherwise flat regions.Furthermore, the image 902 has been downsampled so the edges betweendifferent regions have been blurred.

FIG. 9b shows an image 904 which is a representation of the firstexample image 902 when it is output from the processing module 210. Itcan be seen that almost all of the defective pixels have been fixed. Inparticular, all single and double defective pixels have been fixed andmost triple and quad defective pixel groups have been fixed. It is alsoapparent that the noise in the flat regions of the image has beenreduced. The edges between the different coloured regions have beensharpened.

FIG. 10a shows a representation of a second example image 1002 which isreceived at the processing module 210. The image 1002 has also beenartificially generated as a test image so that it has a large number ofdefective pixels (again, approximately one in every 300 pixels isdefective in the image 1002). The image 1002 has random noise applied toit, which can be seen in the ‘speckled’ nature of the otherwise flatregions. Furthermore, the image 1002 has been downsampled so the edgesbetween different regions have been blurred. It is further noted thatthe edges in image 1002 are circular so they are at differentorientations at different positions in the image.

FIG. 10b shows an image 1004 which is a representation of the secondexample image 1002 when it is output from the processing module 210. Itcan be seen that almost all of the defective pixels have been fixed.Again, all single and double defective pixels have been fixed and mosttriple and quad defective pixel groups have been fixed. It is alsoapparent that the noise in the flat regions of the image has beenreduced. The edges between the different coloured regions have beensharpened, at all edge orientations.

The examples shown in FIGS. 9a to 10b illustrate how effective theprocessing can be. Furthermore, this is achieved with a singleprocessing module to implement defective pixel detection, defectivepixel correction, denoising and sharpening, rather than with separateprocessing modules for each separate function. Furthermore, the kernelused in the processing module is a 5×5 kernel in the example used toprocess the images in FIGS. 9a to 10b , which is smaller than may havebeen used previously in one processing module implementing one of theprocessing functions. Implementing fewer and smaller processing modulesleads to smaller silicon area and lower power consumption (asfunctionality is not replicated in numerous processing modules), and thedifferent processing functions do not make contradictory decisions, sothe resulting images are better and tuning is simpler. This is achievedapplying the Bayesian principle to the grouping process as describedabove, such that knowledge of the grouping is carried over from oneiteration to the next iteration.

In the examples shown in FIGS. 4 and 5 the subsets of pixel values 406and 506 are 5×5 blocks of pixel values. In other examples, the subsetsmay have different sizes.

Furthermore, in the examples described above, the subsets of pixelvalues are square, e.g. kernels 406 and 506 are square. However, inother examples, the subsets of pixel values may be non-square blocks ofpixel values. For example, it may be beneficial to use subsets whichrepresent blocks of pixel values which are wider than they are tall.Just to give an example, subsets could be 3×5 blocks of pixel values(i.e. three rows and five columns). Since the line store module 208stores rows of data values, (e.g. as they arrive in raster scan order),a reduction in the number of rows means that the line store module 208can have a reduced size, and a delay incurred in waiting for the linestore module 208 to receive enough pixel values for the processingmodule 210 to process pixel values can be reduced. Whilst it may beadvantageous to reduce the number of rows in the subsets for the reasonsgiven above, the same is not true for reducing the number of columns inthe subsets (when the data values are received in rows, e.g. in rasterscan order or Boustrophedon order), so it can be beneficial to havesubsets which have more columns than rows (i.e. they are wider than theyare tall). This allows the subsets to retain a large overall size,whilst reducing the number of rows in the subsets. The size and shape ofthe subsets may be designed in dependence on the processing functionsperformed by the processing module 210.

In the examples described above, the subset of pixel values for aparticular pixel value includes and surrounds the particular pixelvalue, e.g. the subset is centred on the particular pixel value. Pixelvalues which are on, or near, the edge of the image may be handled indifferent ways. For example, the size and/or shape of the kernel (or“subset”) for a pixel on or near the image edge could be modified toaccount for the edge of the image. Alternatively, the image could beextended (e.g. by reflection of existing pixel values across the imageedge) such that the size and shape of the kernel (or “subset”) for apixel on or near the image edge does not need to be modified. Thetreatment of edge pixels can vary in different implementations.

In the methods described above with reference to FIG. 3, after the datavalues are received at the processing module, there is a “start-up”procedure (including steps S306 to S314) for the first subset (which canbe considered to be for index i=0), and then there is a “recurrent”procedure (including steps S318 to S330) which are performed forsubsequent subsets (e.g. for i≥1). As described above, the recurrentprocedure is much simpler to implement than the start-up procedure. Insome examples, the start-up procedure is not performed, such that therecurrent procedure can be performed for each of the subsets. This issimpler to implement, and means that the same procedure is performed forall of the pixel values, so the hardware can be simplified. Therecurrent procedure could start with the first pixel in the image, e.g.using a single group with the average value (μ) set to 0.5. This maycause artifacts in the first one or more pixels to be processed, but itwould be simple to implement. Alternatively, the recurrent procedurecould start one or more pixels (e.g. two pixels) before (e.g. to theleft of) the first pixel to be processed in the image, e.g. by makinguse of the reflected region of the image extending beyond the edges ofthe image. Starting with a single group with μ=0.5, would mean that thefirst iteration has a single group into which all of the pixel valuesare classified, and the group indication data (e.g. average of the pixelvalues) for the group may be carried forward to the second iteration. Inthe second iteration, multiple groups, e.g. three groups, may be defined(e.g. a first group surrounding the average value carried forward fromfirst iteration, a group for values below the first group, and a groupfor values above the first group). Pixel values can be classified intothe three groups, and the group indication data (e.g. averages of thepixel values) for the three groups can be carried forward to the thirditeration. The third iteration may be for the first “real” (i.e.non-reflected) pixel to be processed, e.g. the top left pixel of theimage. The recurrent procedure defined above (e.g. steps S318 to S332)can then be performed for the first pixel in the image based on thegroup indication data (e.g. the three averages of the pixel values) forthe groups, and then the process can proceed as described above.

So the start-up procedure (i.e. steps S306 to S314) is not necessarilyperformed in all examples. In examples in which initial iterations areperformed for reflected pixels which do not lie in the image, theseinitial iterations can be performed during the horizontal blankinginterval, such that there is little or no perceptible latency caused byperforming the initial iterations. In some examples, more than twoinitial iterations (or just one initial iteration) may be performedbefore iterations are performed for real pixel values in the image. Bymaking use of image reflection, the group indication data for the groupswill become well trained by the time data is processed for real pixelvalues of the image. These initial iterations for reflected pixels,might only be performed prior to performing an iteration for the firstpixel of the image (e.g. the top left pixel of the image, when the imageis processed in raster scan order). For subsequent rows of pixels, theiteration for the left-most pixel of a row can use the group indicationdata stored for the left-most pixel of the line above. This means thatthe group indication data for the left-most pixel of a line may bestored in the store 212 at least until it is used for processing theleft-most pixel of the line below.

The examples given above are described in terms of an image processingsystem 204 which processes pixel values. The same principles can applyto other types of data processing system, e.g. where data values are notpixel values.

For example, the stream of data values may represent a one-dimensionalarray of data values. In this case, each of the subsets of data valueswithin the stream of data values represents a contiguous block of datavalues within the one-dimensional array.

For example, a 1D stream of data values could be used if the dataprocessing system is an audio processing system and the data values areaudio samples of an audio signal. For example, the audio signal could bereceived and sampled at a microphone and the audio samples can be passedto the audio processing system. The audio processing system may performsome processing functions, some of which may implement a clusteringprocess according to the Bayesian approach as described herein, wherebygroup indication data is stored in one iteration for use in defining aset of groups in a subsequent iteration when processing the next datavalue in the stream. Processed audio samples of the audio signal can beoutputted, e.g. via a speaker.

In another example, a 1D stream of data values could be used if the dataprocessing system is a signal processing system and the data values aresignal samples of a transmitted signal. For example, the signal could bereceived over a wired or wireless channel, at a device in which thesignal processing system is implemented. The device can sample thereceived signal and then the signal processing system may perform someprocessing functions, some of which may implement a clustering processaccording to the Bayesian approach as described herein whereby groupindication data is stored in one iteration for use in defining a set ofgroups in a subsequent iteration when processing the next data value inthe stream. The processed signal samples can be used to extract the datafrom the received signal.

In the examples described above, a pixel value may be replaced by anaverage of a plurality of pixel values. For example, a pixel value isreplaced by an average value of one or more of the maintained groups ifthe pixel value is determined to be representative of a defective pixel,i.e. if the pixel value is not in one of the maintained groups.Furthermore, if a pixel is within one of the maintained groups (e.g. ifit is determined not to be representative of a defective pixel) thendenoising may be applied in which the pixel value is replaced by anaverage value of the group in which it is located. When pixel values arereplaced by averages of multiple pixel values, dynamic range enhancementcan be applied whereby the number of bits used to represent the pixelvalue can be increased. The average value could be, for example, themean, a weighted mean or a mean of a trimmed set of the pixel values inthe group (e.g. the mean of the interquartile of pixel values in agroup) to give just some examples of average values which could be used.By determining an average of pixel values which have been classifiedinto the same group, the number of bits of accuracy of the average valuecan be greater than the number of bits of each individual pixel value.This is because the averaging process reduces the noise. In other words,the averaging process increases the signal to noise ratio (SNR). Thismeans that the SNR of the pixel values output from the processing module210 may be greater than the SNR of the pixel values captured by an imagesensor (e.g. the image sensor 202) and provided to the image processingsystem 204. Therefore, the average value may be determined with agreater number of bits than the pixel value which it is to replace.Increasing the number of bits in this manner increases the dynamic rangeof the pixel values. The clustering approach allows the average value tobe determined from a group of pixel values which are similar to thepixel value to be replaced (similar in terms of location, i.e. they areall within the same subset of pixel values, and similar in terms ofpixel values because they have been grouped into the same group).Therefore, the averaging reduces the random noise in a deterministicway, thereby improving the signal to noise ratio. For example, randomerrors which may be reduced include errors resulting from analogue todigital conversion (ADC noise), thermal noise, quantisation noise, powerplane noise and random Gaussian noise, to give some examples. Averagingover the whole kernel would blur or soften edges within the kernel.However, by averaging over the pixel values within a single group ofsimilar pixel values, the edges between groups within the kernel are notdamaged. The outputted pixel values (i.e. the averaged pixel values)have more bits per pixel than the incoming pixel values, and theoutputted bits have better SNR than the incoming pixel values.

More generally, a dynamic range enhancement method could be appliedwithout necessarily being part of the example method shown in FIG. 8.For example, FIG. 11 illustrates a flow chart for a method ofimplementing a dynamic range enhancement which may be performed by theprocessing module 210 using the clustering approach described above. Instep S1102 pixel values are received at the processing module 210. Eachof the received pixel values has a first number of bits, e.g. eachreceived pixel value may have b_(input) bits where to give just someexamples, b_(input) may be 8, 10 or 12, etc. In step S1104 an index i isset to zero. In step S1106 a set of groups are defined into which pixelvalues of the i^(th) subset can be grouped. The i^(th) subset may forexample include the pixel values within a kernel (e.g. the kernel 406 orthe kernel 506), which may be centred on the i^(th) pixel value. Thegroups may be defined as described in the examples above, e.g. in stepS306 or in step S320. In step S1108 each of the data values within thei^(th) subset is classified into one of the defined groups. As describedabove, the classification of pixel values into groups may be based on acomparison of the pixel value with value ranges defined for each of thegroups, or may be based on detecting one or more edges within the kerneland classifying the pixel values based on the positions of the pixelvalues within the kernel relative to the detected edge(s).

In step S1110 an average value of the pixel values within the i^(th)subset which are classified into one of the groups is determined. Forexample, step S1110 may involve determining an average value of thepixel values which have been classified into the same group as thei^(th) pixel value (i.e. the pixel value at the centre of the i^(th)subset). As described above, the average value could be, for example,the mean, a weighted mean or a mean of a trimmed set of the pixel valuesin the group (e.g. the mean of the interquartile of pixel values in agroup) to give just some examples of average values which could be used.The determined average value has a second number of bits, b_(average),which is greater than the first number of bits, b_(input), i.e.b_(average)>b_(input). For example, b_(average) could be one, two, threeor four greater than b_(input), just to give some examples. As explainedabove, more bits can be used for the average value (compared to thenumber of bits of the input pixel values) because averaging the pixelvalues over a particular group of similar pixel values increases thesignal to noise ratio of the values. By using more bits for the averagevalue, the average value has a greater precision than the input pixelvalues. Since the averaging is performed over similar pixel values (i.e.pixel values classified into the same group), the averaging does notblur the edges in the image.

In step S1112 the i^(th) pixel value is replaced based on the determinedaverage value. For example, the i^(th) pixel value may be replaced withthe determined average value.

In step S1114 the processed i^(th) pixel value is outputted, e.g. fromthe processing module 210 for use in the image processing system 204.The processed pixel value may be used by other processing modules 214within the image processing system 204, and/or outputted from the imageprocessing system 204.

In step S1116 the processing module 210 determines whether there are anymore pixel values to process, and if there are, then the method passesto step S1118 in which the index i is incremented. The method thenpasses back to step S1106 and the method repeats by performing stepsS1106 to S1116 for the next pixel value. If it is determined in stepS1116 that there are no more pixel values to process then the methodends in step S1120.

In the method shown in FIG. 11 each processed pixel value is outputtedas it has been determined (in step S1114). It would be possible tooutput groups of processed pixel values in batches when a whole batch ofpixel values has been processed. For example, when a row of pixel valueshave been determined then all of the processed pixel values in that rowcould be outputted together. Furthermore, the method could wait untilall of the pixel values within an image have been processed before anyof them are outputted.

According to the method shown in FIG. 11, the number of bits of anoutputted pixel value is greater than the number of bits of an inputpixel value. Therefore, the dynamic range of the pixel values can beincreased. For example, each of the outputted pixel values may have oneor two more bits than the corresponding input pixel values. Based on atypical group having 8 or 16 pixel values within it, according to thecentral limit theorem, 1.5 to 2 more bits of accuracy can be achieved byaveraging the pixel values. The averaging process described above allowsthe amount of data in an output pixel value can be greater than theamount of data in the corresponding input pixel value becauseinformation is shared over multiple pixel values.

It is worth noting that this is not the same as super-resolution whichgenerates a higher density of pixels than the source image, because withthe dynamic range enhancement described herein, the same pixel densityis maintained but the number of useful bits per pixel for each pixel isincreased.

It is further worth noting that this is not the same as temporaldenoising, which operates in the temporal domain, rather than thespatial domain (as in the averaging process described above), though theaccuracy, effect and impact of temporal denoising will be improvedwhenever the spatial noise reduction is enhanced.

An image processing system is described herein and shown in FIG. 2 ascomprising a number of functional blocks. This is schematic only and isnot intended to define a strict division between different logicelements of such entities. Each functional block may be provided in anysuitable manner. It is to be understood that intermediate valuesdescribed herein as being formed by an image processing system need notbe physically generated by the image processing system at any point andmay merely represent logical values which conveniently describe theprocessing performed by the image processing system between its inputand output.

The processing systems described herein (i.e. the image processingsystems and data processing systems) may be embodied in hardware on anintegrated circuit. The processing systems described herein may beconfigured to perform any of the methods described herein. Generally,any of the functions, methods, techniques or components described abovecan be implemented in software, firmware, hardware (e.g., fixed logiccircuitry), or any combination thereof. The terms “module,”“functionality,” “component”, “element”, “unit”, “block” and “logic” maybe used herein to generally represent software, firmware, hardware, orany combination thereof. In the case of a software implementation, themodule, functionality, component, element, unit, block or logicrepresents program code that performs the specified tasks when executedon a processor. The algorithms and methods described herein could beperformed by one or more processors executing code that causes theprocessor(s) to perform the algorithms/methods. Examples of acomputer-readable storage medium include a random-access memory (RAM),read-only memory (ROM), an optical disc, flash memory, hard disk memory,and other memory devices that may use magnetic, optical, and othertechniques to store instructions or other data and that can be accessedby a machine.

The terms computer program code and computer readable instructions asused herein refer to any kind of executable code for processors,including code expressed in a machine language, an interpreted languageor a scripting language. Executable code includes binary code, machinecode, bytecode, code defining an integrated circuit (such as a hardwaredescription language or netlist), and code expressed in a programminglanguage code such as C, Java or OpenCL. Executable code may be, forexample, any kind of software, firmware, script, module or librarywhich, when suitably executed, processed, interpreted, compiled,executed at a virtual machine or other software environment, cause aprocessor of the computer system at which the executable code issupported to perform the tasks specified by the code.

A processor, computer, or computer system may be any kind of device,machine or dedicated circuit, or collection or portion thereof, withprocessing capability such that it can execute instructions. A processormay be any kind of general purpose or dedicated processor, such as aCPU, GPU, System-on-chip, state machine, media processor, anapplication-specific integrated circuit (ASIC), a programmable logicarray, a field-programmable gate array (FPGA), or the like. A computeror computer system may comprise one or more processors.

It is also intended to encompass software which defines a configurationof hardware as described herein, such as HDL (hardware descriptionlanguage) software, as is used for designing integrated circuits, or forconfiguring programmable chips, to carry out desired functions. That is,there may be provided a computer readable storage medium having encodedthereon computer readable program code in the form of an integratedcircuit definition dataset that when processed (i.e. run) in anintegrated circuit manufacturing system configures the system tomanufacture an image processing system or data processing systemconfigured to perform any of the methods described herein, or tomanufacture an image processing system or data processing systemcomprising any apparatus described herein. An integrated circuitdefinition dataset may be, for example, an integrated circuitdescription.

Therefore, there may be provided a method of manufacturing, at anintegrated circuit manufacturing system, an image processing system ordata processing system as described herein. Furthermore, there may beprovided an integrated circuit definition dataset that, when processedin an integrated circuit manufacturing system, causes the method ofmanufacturing an image processing system or data processing system to beperformed.

An integrated circuit definition dataset may be in the form of computercode, for example as a netlist, code for configuring a programmablechip, as a hardware description language defining an integrated circuitat any level, including as register transfer level (RTL) code, ashigh-level circuit representations such as Verilog or VHDL, and aslow-level circuit representations such as OASIS® and GDSII. Higher levelrepresentations which logically define an integrated circuit (such asRTL) may be processed at a computer system configured for generating amanufacturing definition of an integrated circuit in the context of asoftware environment comprising definitions of circuit elements andrules for combining those elements in order to generate themanufacturing definition of an integrated circuit so defined by therepresentation. As is typically the case with software executing at acomputer system so as to define a machine, one or more intermediate usersteps (e.g. providing commands, variables etc.) may be required in orderfor a computer system configured for generating a manufacturingdefinition of an integrated circuit to execute code defining anintegrated circuit so as to generate the manufacturing definition ofthat integrated circuit.

An example of processing an integrated circuit definition dataset at anintegrated circuit manufacturing system so as to configure the system tomanufacture an image processing system will now be described withrespect to FIG. 12.

FIG. 12 shows an example of an integrated circuit (IC) manufacturingsystem 1202 which is configured to manufacture an image processingsystem as described in any of the examples herein. In particular, the ICmanufacturing system 1202 comprises a layout processing system 1204 andan integrated circuit generation system 1206. The IC manufacturingsystem 1202 is configured to receive an IC definition dataset (e.g.defining an image processing system as described in any of the examplesherein), process the IC definition dataset, and generate an IC accordingto the IC definition dataset (e.g. which embodies an image processingsystem as described in any of the examples herein). The processing ofthe IC definition dataset configures the IC manufacturing system 1202 tomanufacture an integrated circuit embodying an image processing systemas described in any of the examples herein.

The layout processing system 1204 is configured to receive and processthe IC definition dataset to determine a circuit layout. Methods ofdetermining a circuit layout from an IC definition dataset are known inthe art, and for example may involve synthesising RTL code to determinea gate level representation of a circuit to be generated, e.g. in termsof logical components (e.g. NAND, NOR, AND, OR, MUX and FLIP-FLOPcomponents). A circuit layout can be determined from the gate levelrepresentation of the circuit by determining positional information forthe logical components. This may be done automatically or with userinvolvement in order to optimise the circuit layout. When the layoutprocessing system 1204 has determined the circuit layout it may output acircuit layout definition to the IC generation system 1206. A circuitlayout definition may be, for example, a circuit layout description.

The IC generation system 1206 generates an IC according to the circuitlayout definition, as is known in the art. For example, the ICgeneration system 1206 may implement a semiconductor device fabricationprocess to generate the IC, which may involve a multiple-step sequenceof photo lithographic and chemical processing steps during whichelectronic circuits are gradually created on a wafer made ofsemiconducting material. The circuit layout definition may be in theform of a mask which can be used in a lithographic process forgenerating an IC according to the circuit definition. Alternatively, thecircuit layout definition provided to the IC generation system 1206 maybe in the form of computer-readable code which the IC generation system1206 can use to form a suitable mask for use in generating an IC.

The different processes performed by the IC manufacturing system 1202may be implemented all in one location, e.g. by one party.Alternatively, the IC manufacturing system 1202 may be a distributedsystem such that some of the processes may be performed at differentlocations, and may be performed by different parties. For example, someof the stages of: (i) synthesising RTL code representing the ICdefinition dataset to form a gate level representation of a circuit tobe generated, (ii) generating a circuit layout based on the gate levelrepresentation, (iii) forming a mask in accordance with the circuitlayout, and (iv) fabricating an integrated circuit using the mask, maybe performed in different locations and/or by different parties.

In other examples, processing of the integrated circuit definitiondataset at an integrated circuit manufacturing system may configure thesystem to manufacture an image processing system without the ICdefinition dataset being processed so as to determine a circuit layout.For instance, an integrated circuit definition dataset may define theconfiguration of a reconfigurable processor, such as an FPGA, and theprocessing of that dataset may configure an IC manufacturing system togenerate a reconfigurable processor having that defined configuration(e.g. by loading configuration data to the FPGA).

In some embodiments, an integrated circuit manufacturing definitiondataset, when processed in an integrated circuit manufacturing system,may cause an integrated circuit manufacturing system to generate adevice as described herein. For example, the configuration of anintegrated circuit manufacturing system in the manner described abovewith respect to FIG. 12 by an integrated circuit manufacturingdefinition dataset may cause a device as described herein to bemanufactured.

In some examples, an integrated circuit definition dataset could includesoftware which runs on hardware defined at the dataset or in combinationwith hardware defined at the dataset. In the example shown in FIG. 12,the IC generation system may further be configured by an integratedcircuit definition dataset to, on manufacturing an integrated circuit,load firmware onto that integrated circuit in accordance with programcode defined at the integrated circuit definition dataset or otherwiseprovide program code with the integrated circuit for use with theintegrated circuit.

The implementation of concepts set forth in this application in devices,apparatus, modules, and/or systems (as well as in methods implementedherein) may give rise to performance improvements when compared withknown implementations.

The performance improvements may include one or more of increasedcomputational performance, reduced latency, increased throughput, and/orreduced power consumption. During manufacture of such devices,apparatus, modules, and systems (e.g. in integrated circuits)performance improvements can be traded-off against the physicalimplementation, thereby improving the method of manufacture. Forexample, a performance improvement may be traded against layout area,thereby matching the performance of a known implementation but usingless silicon. This may be done, for example, by reusing functionalblocks in a serialised fashion or sharing functional blocks betweenelements of the devices, apparatus, modules and/or systems. Conversely,concepts set forth in this application that give rise to improvements inthe physical implementation of the devices, apparatus, modules, andsystems (such as reduced silicon area) may be traded for improvedperformance. This may be done, for example, by manufacturing multipleinstances of a module within a predefined area budget.

The applicant hereby discloses in isolation each individual featuredescribed herein and any combination of two or more such features, tothe extent that such features or combinations are capable of beingcarried out based on the present specification as a whole in the lightof the common general knowledge of a person skilled in the art,irrespective of whether such features or combinations of features solveany problems disclosed herein. In view of the foregoing description itwill be evident to a person skilled in the art that variousmodifications may be made within the scope of the invention.

What is claimed is:
 1. An image processing system configured to processa stream of pixel values, the image processing system comprising: aprocessing module configured to implement processing of a particularpixel value of the stream by operating on a particular subset of pixelvalues of the stream, by: classifying each of the pixel values withinthe particular subset into a group of a set of groups; processing theparticular pixel value using one or more of the pixel values of theparticular subset in dependence on the classification of the pixelvalues of the particular subset into the groups, wherein said processingthe particular pixel value comprises performing, in a consolidatedoperation, multiple processing functions which depend upon theclassification of pixel values of the particular subset into the groups,such that the classification of the pixel values within the particularsubset into the groups is consistent for said multiple processingfunctions; and outputting the processed particular pixel value.
 2. Theimage processing system of claim 1, wherein the multiple processingfunctions comprise two or more of: (i) defective pixel detection, (ii)defective pixel correction, (iii) denoising, (iv) sharpening, and (v)de-mosaicing.
 3. The image processing system of claim 1, wherein theimage processing system is implemented in a camera pipeline.
 4. Theimage processing system of claim 1, further comprising a storeconfigured to store group indication data which indicates one or moregroups into which pixel values can be grouped, wherein the processingmodule is configured to process, in each of a plurality of iterations, arespective particular pixel value of the stream by operating on arespective particular subset of pixel values of the stream, by, in eachof the plurality of iterations: retrieving, from the store, groupindication data for at least one group; using the retrieved groupindication data to define the set of groups into which pixel valueswithin the particular subset can be grouped; performing said: (i)classifying each of the pixel values within the particular subset into agroup of a set of groups, and (ii) processing the particular pixel valueusing one or more of the pixel values of the particular subset independence on the classification of the pixel values of the particularsubset into the groups; and storing, in the store, group indication datafor at least one group of the set of groups, for use in a subsequentiteration.
 5. The image processing system of claim 4, wherein saidstoring group indication data for at least one group comprises storing,as an indicative value for a group, an average value of the pixel valueswhich are classified as being part of that group.
 6. The imageprocessing system of claim 4, wherein said storing group indication datafor at least one group comprises storing at least one of: (i) anindication of the spread of the data values which are classified asbeing part of that group, (ii) an indicative position of the data valueswhich are classified as being part of that group, and (iii) anindication of the number of members of the group.
 7. The imageprocessing system of claim 1, wherein each of the pixel values withinthe particular subset is classified into one of the groups of the set ofgroups based on the value of that pixel value.
 8. The image processingsystem of claim 1, configured to detect an edge within the particularsubset, wherein each of the pixel values within the particular subset isclassified into one of the groups of the set of groups based on therelative position of the pixel value compared to the detected edge. 9.The image processing system of claim 1, wherein group indication datafor a group indicates a range of pixel values which are to be classifiedinto the group.
 10. The image processing system of claim 9, wherein thegroup indication data for a group comprises an indicative value withinthe range of pixel values which are to be classified into the group. 11.The image processing system of claim 9, wherein the group indicationdata for a group comprises indications of the upper and lower bounds ofthe range of pixel values which are to be classified into the group. 12.The image processing system of claim 1, wherein the stream of pixelvalues represents an image as a two-dimensional array of pixel values,and wherein each of the subsets of pixel values within the stream ofdata values represents a contiguous block of pixel values within thetwo-dimensional array.
 13. The image processing system of claim 12,wherein a particular subset of pixel values represents a block of pixelvalues which includes and surrounds the corresponding particular pixelvalue.
 14. The image processing system of claim 1, wherein the imageprocessing system is configured to receive lines of pixel values, andwherein the image processing system comprises a line store moduleconfigured to store lines of pixel values such that the subsets of pixelvalues can span over more than one line.
 15. The image processing systemof claim 14, wherein the lines of pixel values are rows of pixel values,wherein the image processing system is configured to receive the streamof pixel values according to a raster scan order, and wherein thesubsets of pixel values represent blocks of pixel values within the twodimensional array which are wider than they are tall.
 16. The imageprocessing system of claim 1, wherein the set of groups comprises up ton groups, and wherein the processing module is further configured to:after the classification of the pixel values of the particular subsetinto the groups, and prior to said processing the particular pixelvalue, selectively discard one or more of the groups based on the numberof pixel values which are classified into each of the groups, such thatup to m groups are maintained, where n≥m.
 17. The image processingsystem of claim 16, wherein said processing the particular pixel valueusing one or more of the pixel values of the particular subset independence on the classification of the pixel values of the particularsubset into the groups comprises: if the particular pixel value iswithin a maintained group, determining that the particular pixel valueis not representative of a defective pixel; and if the particular pixelvalue is not within a maintained group, determining that the particularpixel value is representative of a defective pixel.
 18. The imageprocessing system of claim 1, wherein the processing module isconfigured to: determine, for each of at least one group of the set ofgroups, a centroid representative of a position of the group; anddetermine, for each of the at least one group of the set of groups, anaverage value of the pixel values within the group.
 19. A method ofprocessing a stream of pixel values in an image processing system, themethod comprising: implementing processing of a particular pixel valueof the stream by operating on a particular subset of pixel values of thestream, by: classifying each of the pixel values within the particularsubset into a group of a set of groups; processing the particular pixelvalue using one or more of the pixel values of the particular subset independence on the classification of the pixel values of the particularsubset into the groups, wherein said processing the particular pixelvalue comprises performing, in a consolidated operation, multipleprocessing functions which depend upon the classification of pixelvalues of the particular subset into the groups, such that theclassification of the pixel values within the particular subset into thegroups is consistent for said multiple processing functions; andoutputting the processed particular pixel value.
 20. A non-transitorycomputer readable storage medium having stored thereon a computerreadable description of an integrated circuit that, when processed in anintegrated circuit manufacturing system, causes the integrated circuitmanufacturing system to manufacture an image processing systemconfigured to process a stream of pixel values, the image processingsystem comprising: a processing module configured to implementprocessing of a particular pixel value of the stream by operating on aparticular subset of pixel values of the stream, by: classifying each ofthe pixel values within the particular subset into a group of a set ofgroups; processing the particular pixel value using one or more of thepixel values of the particular subset in dependence on theclassification of the pixel values of the particular subset into thegroups, wherein said processing the particular pixel value comprisesperforming, in a consolidated operation, multiple processing functionswhich depend upon the classification of pixel values of the particularsubset into the groups, such that the classification of the pixel valueswithin the particular subset into the groups is consistent for saidmultiple processing functions; and outputting the processed particularpixel value.